Rendering of photorealistic computer graphics images

ABSTRACT

A method, apparatus, and system are provided for rendering of photorealistic computer graphics images. According to one embodiment, an image is partitioned into regions, each of the regions having a discontinuity edge, a boundary, edge pixels along the boundary, and remaining pixels, and orientation of the discontinuity edge is estimated by computing a direction of least discrepancy within each of the regions by evaluating the edge pixels along the boundary of each of the regions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of a prior applicationSer. No. 09/303,966, filed on May 03, 1999.

FIELD OF THE INVENTION

The present invention relates generally to the field of computer imagingand more particularly, to rendering of photorealistic computer graphicimages.

BACKGROUND OF THE INVENTION

The ability to synthesize photorealistic images in real-time has beenthe goal of practitioners in the field of computer graphics for manyyears. A variety of different techniques are known for generating imagesof three-dimensional objects on a computer graphics display. One classof techniques divide a two-dimensional array of data values into shadedpolygons, which are then projected onto the display screen. Toaccelerate the image generation process, many polygon-based techniquesutilize a special graphics processor to alleviate the computationalburden on the computer's central processing unit (CPU).

Another class of computer imaging techniques is known as ray tracing.Ray tracing is a pixel-based technique that is capable of producinghighly realistic images in computer graphic systems. A chief drawback ofray tracing techniques, however, is the extensive computations requiredto generate each pixel of the display screen. These intensivecomputations often impose a severe burden on the computer processinghardware. The slow processing times associated with ray tracingtechniques have limited their application in computer graphics systems.For example, an article entitled, “Outlook on Computer Graphics”, by D.P. Greenburg, IEEE Computer, 31(1): 36-36 (January 1998), suggests thatit will not be until the year 2025 before computer systems have thedisplay and computational capability to produce realistic, real-timeimages using pixel-based techniques.

An example of a computer system that utilizes ray tracing is describedin “Antialiased Ray tracing by Adaptive Progressive Refinement,” by J.Painter and K. Sloan, Computer Graphics (SIGGRAPH '89 Proceedings), Vol.23, pages 281-288 (July 1989). Further background in this area may befound in U.S. Pat. No. 5,872,902, which teaches a hardwareimplementation of a computationally intensive anti-aliasing techniquefor generating three-dimensional images on a workstation graphicsprocessor. U.S. Pat. No. 5,831,623 discloses a volume renderingapparatus for visualizing an image on a display screen of an imagingdevice such as a computer tomagraphy scanner for a magnetic resonanceimaging machine. A method and system for generating an anti-aliasingimage of a three-dimensional surface is also described in U.S. Pat. No.5,542,032, which teaches performing certain floating-point arithmeticand comparison operations on pixel data.

Despite the rapidly increasing power of computers, global illuminationis far from being a real-time process. Accurate radiance evaluationsoften require hours of computation for complex scenes. To balancerendering speed and visual realism, global illumination algorithms haveoften adopted a progressive refinement approach, like that described inthe Painter and Sloan article mentioned above. Progressive refinementmethods typically sample densely where sharp features are identified. Inareas of the image plane where there is an absence of sharpfeatures—i.e., the image data changes slowly—progressive refinementtechniques sample very sparsely, and then interpolate.

The problem with these past techniques is that image artifacts are oftenlost when the sampling criteria is minimized. For instance, if it isdesired to keep the sampling rate below 10%, many prior art progressiverefinement approaches prove to be inadequate. In other words, althoughsuch techniques provide a reasonable approach to the problem, theyrequire relatively high sampling rates to provide fast rendering ofphotorealistic computer graphics images. At low sampling rates (e.g.,less than 10%) previous techniques such as adaptive stochastic samplingsuffer from artifacts including heavily jagged edges, missing objectparts, and missing high-frequency details.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and notlimitation, in the figures of the accompanying drawings, where:

FIG. 1 conceptually illustrates the rendering pipeline, according to oneembodiment.

FIGS. 2A-2D conceptually illustrate examples illustrating the pixel onprocess as controlled by the DCM accelerator, according to oneembodiment.

FIGS. 3A-3B conceptually illustrate examples of different discontinutiespresent within the image data of a block, according to one embodiment.

FIGS. 4A-4C conceptually illustrate various aspects of the constructionof the finite element approximate, according to one embodiment.

FIG. 5 conceptually illustrates an example of an oriented finite elementfor a certain discontinuity orientation within a pixel block, accordingto one embodiment.

DETAILED DESCRIPTION

Throughout the following description, numerous specific details are setforth in order to provide a thorough understanding of the embodiments ofthe present invention. However, it should be understood that theembodiment of the present invention could be practiced without theseparticulars. In other instances, well known elements have not been shownor described in detail to avoid unnecessarily obscuring the embodimentsof the present invention.

A new technology for fast rendering of photorealistic computer graphicsimages is presented. In the context of the present application this newtechnology is referred to as a directional coherence map (DCM). DCM is apixel-based, rather than a polygon-based, technology capable ofaccurately reproducing subtle shading, shadow, and inter-reflectioneffects that are commonly seen in real-world scenes. As will becomeapparent to those of ordinary skill in the art, the DCM acceleratorsignificantly speeds up rendering of ray-traced displays as compared toconventional techniques. In addition, DCM technology may be implementedin a variety of personal computer graphics hardware.

Overview of the DCM Accelerator

The DCM technique for handling general radiance discontinuities in aprogressive ray tracing framework includes two primary components:first, an adaptive partitioning of the image plane into blocks, suchthat each block includes at most one discontinuity edge. Secondly, anestimation of the orientation of the discontinuity edge in each block.The estimation is achieved by computing the “direction of leastdiscrepancy” within the block. The DCM assists in capturing radiancediscontinuities by way of finite element approximation to the radiancefunction, with the finite elements on each block being oriented inaccordance with the orientation of the discontinuity within thatparticular block.

According to one embodiment, instead of evaluating all screen pixels—asis done in conventional ray tracing—the DCM accelerator evaluatesthrough ray tracing only the pixels on the boundary of a block. Theboundary or edge pixels typically comprise a small fraction of all ofthe pixels in a display. As discussed previously, one of the maindrawbacks of traditional ray tracing techniques is that they are verycomputationally intensive; this generally prohibits their use inordinary computer graphics systems. However, by evaluating only thepixels on the boundary of each block, the DCM approach is able tocombine generation of a very high quality image with a small samplingsize. By way of example, the DCM technique is capable of renderingphotorealistic computer graphics images by sampling as few as 6% of thepixels that make up the image plane.

Following evaluation of the edge or boundary pixels, the remainingpixels are interpolated from the evaluated pixels through anedge-preserving finite element approximation within each image block.The finite element approximant in each block is oriented according tothe estimated orientation of the discontinuity within the block. Despitethe fact that only a small percentage of all the pixels are evaluatedthrough ray tracing, the quality of the resulting image is very highbecause by preserving discontinuities, the DCM preserves the highfrequency details of the rendered image.

FIG. 1 conceptually illustrates one embodiment of the renderingpipeline, according to one embodiment. As can be seen, the system ofFIG. 1 has two main stages. The first is the regular subdivision stagein which the image plane is partitioned into blocks. To perform theregular subdivision, the entire image plane maybe recursively subdividedit into blocks (e.g., by four). Note that during regular subdivision,the four corner pixels of each block may be sampled, and an approximateimage may be created for display at any time by interpolating the cornervalues.

According to one embodiment, the second stage comprises an iterativeprocess in which the DCM is constructed and refined. A subset of blocksare selected as edge blocks, and then analyzed for discontinuities.Blocks not selected simply go to another step of regular subdivision.For each edge block, the block boundary pixels are densely sampled, andthe block may be subdivided (e.g., into four quads) for the nextiteration. From the evaluated boundary pixels the discontinuities on theedge may be inferred. This information is recorded into the DCM, whereit can be later used to construct an oriented finite elementapproximation of the block. It should be understood that the orientedfinite elements on edge blocks, and the bilinear interpolants on anyother block, may be resampled at any time at user request.

Pixel Evaluation and Discontinuity Estimation

FIGS. 2A-2D conceptually illustrate the pixel evaluation processcontrolled by the DCM accelerator. FIG. 2A simply shows the image plane10, which consists of the data to be operated upon by the computergraphics hardware. Taking a divide-and-conquer approach, the DCM treatsdiscontinuities by partitioning the image plane into small blocks sothat most blocks are crossed by more than one discontinuity edge.

FIG. 2B shows the image plane 10 following adaptive block partitioning.Note that for this example the image plane is first partitioned into a3×3 array of square blocks, which includes individual blocks 11, 12 and13. Blocks 11 and 12 are shown being further subdivided into foursmaller quadrants. In addition, the upper, right quadrant of block 12has been further subdivided into four smaller blocks.

The partitioning example of FIG. 2B produces square blocks. However, itshould be apparent to those of ordinary skill that other types of blocksmay also be generated during partitioning. Furthermore, the extent ofthe partitioning is dependent upon the computational power that isavailable. For example, a moving image may not require a great deal ofdetail to be discerned, so larger block sizes may be selected.Conversely, if the image to be viewed on the display is a static image,smaller block sizes may be selected to resolve more details of theimage; thus leading to further subdivision and partitioning of the imageplane.

In one embodiment, an elementary block size is 8×8 pixels, but thisnumber could be smaller (e.g., 4×4) or larger in accordance with theabove discussion.

FIG. 2C conceptually illustrates boundary evaluation of a single block13 of image plane 10. The thick, heavy boundary line 14 of block 13 inFIG. 2C represents the edge pixels on the block boundary. According toone embodiment, it is only these edge pixels that are evaluated as abasis for providing a best estimation of the sharp edge direction forthe block. The sharp edge direction is referred to as the direction ofleast discrepancy and is illustrated in FIG. 2C by arrow 15. Another wayto conceptualize the direction of least discrepancy is that it refers tothe direction in which the image is changing slowest within the block.

At this point, it may be helpful to refer to FIGS. 3A and 3B, whichconceptually illustrate examples of different discontinuities presentwithin the image data of a block, according to one embodiment. Note thateach of the blocks of FIGS. 3A and 3B include a shaded portion and anunshaded portion. For example, in FIG. 3A, region 22 is unshaded andregion 21 is shaded. Similarly, in FIG. 3B, region 23 is shaded andregion 24 is unshaded.

With reference once again to FIG. 2C, the orientation of thediscontinuity in block 13 is computed from the boundary pixels 14 as thedirection of least discrepancy 15. It should be understood that thisorientation is only an estimate, since the direction of the actualdiscontinuity could lie anywhere from 0° to 180° (actually 0°-360°considering opposite directions). According to one embodiment, thenumber of possible edge directions is discretized, e.g., into eightdifferent directions. For each of the different directions a discrepancynumber is computed from the evaluated boundary pixels. In oneimplementation, the discrepancy number corresponds to the sum of thedifferences of the pixels on opposing sides of the boundary along aparticular direction. In other words, all of the differences along aparticular direction are first computed by subtracting pixel valueslocated on opposite sides of the boundary. Next, all of the differencesare summed, with the result being the discrepancy number associated withthat particular direction. For each of the discrete number ofdirections, the one direction having the smallest discrepancy is chosenas the direction of least discrepancy.

After one of the directions has been selected as the direction of leastdiscrepancy (i.e., the edge direction) a straightforward bilinearinterpolation is performed oriented along the selected direction. Forexample, FIG. 2D shows interpolation of the remaining interior pixels ofblock 13 through oriented finite element construction. Performing abilinear interpolation along the discontinuity edges of the image leadsto smooth edges in the final reproduced image. Smooth edges, obviously,are an important characteristic of a high quality picture. The resultingapproximate image is produced when all of the blocks have been processedas described above.

Mathematical Description

For an image function ƒ(x), the direction of the least discrepancym_(k)(of a k×k block B_(k)) is defined to be the unit factor thatminimizes the contour integral,${d(n)} = {\frac{1}{s}{\int_{C}{\left( {{f\left( {x + {t_{x}n}} \right)} - {f(x)}} \right)^{2}{s}}}}$

where C is the boundary contour of B_(k) and s is the length of thecontour. Practitioners in the art will appreciate that the integrationactually only needs to extend over half the contour. For a fixeddirection n and a point x on C, the scalar t_(x) is chosen such that theparametric line y(t)=x+tn intersects the contour C at x and y=x+t_(x)n,as is shown in FIG. 4A. Note that in FIG. 4A the boundary or edge pixelsare represented by heavy dark line 45.

For computer implementation, n=[cos Θ, sin Θ] and the angular range0≦Θ<π is discretized into h directions Θ_(i)=i π/h; 0≦i<h−1. For eachdirection n_(i)=[cos Θ_(i), sin Θ_(i)], the directional discrepancyd(n_(i)) is computed as:$d_{i} = {{d\left( n_{i} \right)} = {\frac{1}{4\left( {k - 1} \right)}{\sum\limits_{p \in P}^{\quad}\quad \left( {{f\left( {p + {t_{p}n_{i}}} \right)} - {f(p)}} \right)^{2}}}}$

where P is the set of all pixels in C and t_(p) is chosen such that theline y(t)=p+tn_(i) intersects the contour C at p and p+t_(p)n_(i). Next,the sequence {d₀, . . . , d_(h-l)} is evaluated and the minimumd_(j)=min{d₀ . . . , d_(h-l)} is computed to determine with direction ofleast discrepancy; m(B_(k))=n_(j).

The image function in block B_(k) may be approximated by a finiteelement function oriented along the direction of least discrepancy. Thefinite element approximation is a continuous function consisting ofbilinear elements (i.e., quadratic polynomials).

Oriented Finite Elements

FIGS. 4B and 4C conceptually illustrate the construction of the finiteelement, according to one embodiment. At this point, the direction ofleast discrepancy has been found from the evaluated pixels on theboundary contour C (FIG. 4A). In FIG. 4B, there is shown theconstruction of a typical bilinear element on a quadrilateralQ=[Z₁Z₂Z₃Z₄] with known node valuesƒ_(n) (z_(i)), i=1:4. Essentially,this construction is a Gouraud interpolation with the scan line rotatedto be parallel with the least discrepancy direction. Note that FIG. 4Bis a zoomed version of the shaded element 47 in FIG. 4C. Each of theseillustrations is provided to show a typical bilinear element. By way offurther example, FIG. 5 provides an example of an oriented finiteelement for a different discontinuity orientation for an 8×8 pixelblock.

It will be appreciated that the above description of least discrepancydirection in oriented finite elements may be easily extended to conveximage blocks, including the non-square blocks, which are oftenencountered in a quadtree subdivision of the image plane. Practitionerswill further appreciate that the least discrepancy direction approachprovides beneficial results because of image-space coherence. Coherenceis typically referred to as the degree to which parts of the scene orits projection exhibit local similarities. A discontinuity edgerepresents a break of coherence, since image data changes abruptlyacross the edge. However, discontinuities do not break all forms ofcoherence. Specifically, image data is typically coherent along thedirection of the discontinuity edge even if they change abruptly acrossthe edge. For a block with a simple discontinuity edge, the leastdiscrepancy direction represents the direction of maximal coherence ascan be inferred from the evaluated boundary pixels. By orienting thefinite elements among this direction, the present invention maximizesthe likelihood of capturing the discontinuity edge along with itscharacteristics.

The DCM method, according to one embodiment, provides great advantagesover prior art techniques because it allows the generation of highquality images from a small percentage of evaluated pixels. By capturingand preserving discontinuities, the DCM accelerator also overcomes thefundamental obstacle faced by previous adaptive sampling approaches. Fora global illumination rendering a scene consisting of smooth surfaces,the DCM accelerator of the present invention is capable of producinghigh quality images very efficiently. A typical implementation of thepresent invention can produce a photorealistic computer image byevaluating less than 6% of the pixels. At such a low sampling rate,conventional adaptive sampling approaches suffer from numerous problemsdescribed previously. A personal computer with a DCM accelerator,according to one embodiment, uses 16 times less CPU power than onewithout the DCM accelerator. Thus, personal computer hardwaremanufacturers can implement, according to one embodiment, in 3-Dgraphics rendering pipelines to enable photorealistic rendering atinteractive rates.

It should be understood that although the embodiments of the presentinvention have been described in conjunction with certain specificembodiments, numerous modifications and alterations could be madewithout departing from the scope of the present invention. Accordingly,the specification and drawings are to be regarded in an illustrative,rather than a restrictive sense.

What is claimed is:
 1. An apparatus, comprising: a processor; adirectional coherence map (DCM) accelerator coupled with the processor,the DCM accelerator to: partition an image into a plurality of regions,wherein each of the plurality of regions having a discontinuity edge, aboundary, edge pixels along the boundary, and remaining pixels; andestimate orientation of the discontinuity edge by computing direction ofleast discrepancy within each of the plurality of regions by evaluatingthe edge pixels along the boundary of each of the plurality of regions.2. The apparatus of claim 1, wherein the DCM accelerator is further toreproduce the image.
 3. The apparatus of claim 1, wherein the DCMaccelerator is further to interactively bilinearly interpolate theremaining pixels of each of the plurality of regions, wherein theinteractive bilinear interpolation comprises interpolation orientationalong the direction of least discrepancy.
 4. The apparatus of claim 1,the DCM accelerator is further to construct a finite elementapproximation for each of the plurality of regions in accordance withthe estimated orientation of the discontinuity edge.
 5. The apparatus ofclaim 1, wherein the plurality of regions comprises a plurality ofblocks, each of the plurality of blocks having a plurality of squares ofvarying sizes.
 6. The apparatus of claim 1, wherein the DCM acceleratoris further to partition each of the plurality of regions into aplurality of smaller regions.
 7. The apparatus of claim 1, wherein theDCM accelerator is further to interactively select each of the pluralityof regions.
 8. The apparatus of claim 1, wherein estimate orientationcomprises interactively estimate orientation.
 9. The apparatus of 2,wherein reproduce the image comprises interactively reproduce the image.10. The apparatus of claim 1, wherein the DCM accelerator comprises apixel-based DCM accelerator to reproduce one or more of the following:subtle shading, shadow, and inter-reflection effects.
 11. The apparatusof claim 2, wherein the reproduce the image comprises speedily reproducethe image.
 12. A system, comprising: a storage device; a processorcoupled with the storage device; and a directional coherence map (DCM)accelerator coupled with the processor, the DCM accelerator to:partition an image into a plurality of regions, wherein each of theplurality of regions having a discontinuity edge, a boundary, edgepixels along the boundary, and remaining pixels; and estimateorientation of the discontinuity edge by computing direction of leastdiscrepancy within each of the plurality of regions by evaluating theedge pixels along the boundary of each of the plurality of regions. 13.The system of claim 12, wherein the DCM accelerator is further toreproduce the image.
 14. The system of claim 12, wherein the DCMaccelerator is further to interactively bilinearly interpolate theremaining pixels of each of the plurality of regions, wherein theinteractive bilinear interpolation comprises interpolation orientationalong the direction of least discrepancy.
 15. A method, comprising:partitioning an image into a plurality of regions, each of the pluralityof regions having a discontinuity edge, a boundary, edge pixels alongthe boundary, and remaining pixels; and estimating orientation of thediscontinuity edge by computing a direction of least discrepancy withineach of the plurality of regions by evaluating the edge pixels along theboundary of each of the plurality of regions.
 16. The method of claim15, further comprises reproducing the image.
 17. The method of claim 15,further comprises interactive bilinear interpolation of the remainingpixels of each of the plurality of regions, wherein the interactivebilinear interpolation is oriented along the direction of leastdiscrepancy.
 18. The method of claim 15, wherein the plurality ofregions comprises a plurality of blocks, each block of the plurality ofblocks having a plurality of squares of varying sizes.
 19. The method ofclaim 15, further comprises edge-preserving finite element approximationwithin each of the plurality of regions, wherein the edge-preservingfinite element approximation is in accordance with the estimatedorientation of the discontinuity edge.
 20. The method of claim 15,further comprises recording the estimated orientation of thediscontinuity edge.
 21. The method of claim 20, further comprises usingthe recorded estimated orientation of the discontinuity edge toconstruct an approximate image.